Skip to content

Split server upstream transports#10

Merged
pirate merged 145 commits into
mainfrom
simplify-transports
May 28, 2026
Merged

Split server upstream transports#10
pirate merged 145 commits into
mainfrom
simplify-transports

Conversation

@pirate
Copy link
Copy Markdown
Member

@pirate pirate commented May 26, 2026

Summary

This PR turns the transport/client/server split into explicit owning classes, adds the new topology-aware router path, and brings the translated Python and Go surfaces back into line with the TypeScript source for the files marked MODCDP_TRANSLATE.

The main shape is now:

  • ModCDPClient owns launcher, upstream, injector, router, types, config, and optional server_config as concrete component objects/config objects.
  • AutoSessionRouter owns browser target/session/execution-context routing and Mod.getTopology.
  • CDPTypes owns native CDP + ModCDP + custom command/event schema registration, runtime validation, JSON-schema normalization, and alias hydration.
  • ModCDPServer is TS-only service-worker code that composes a normal upstream ModCDPClient plus a downstream transport set.
  • Python and Go are intentionally limited to the translated Stagehand-needed surface: websocket upstream CDP, local/remote/bb/none launchers, bb/cli/cdp/discover/none injectors, router/config/CDPTypes parity, and real browser tests. TS-only transports and service-worker/server code are marked and not translated.

Architecture changes

Client composition

  • Reworked js/src/client/ModCDPClient.ts from a large option-normalizing client into a component-composed client:
    • launcher: BrowserLauncher
    • upstream: UpstreamTransport
    • injector: ExtensionInjector | null
    • router: AutoSessionRouter
    • types: CDPTypes
    • config: ModCDPClientConfig
    • server_config: ModCDPServerConfig | null
  • Removed the old raw-send/session facade shape from the client. Typed client sends go through client.send(...); raw transport access is on client.upstream.send(...); routed browser sends go through client.router.
  • Moved route config under router.router_routes and server configuration under server_config, matching the same shape that gets sent to Mod.configure.
  • Mod.addCustomCommand, Mod.addCustomEvent, and Mod.addMiddleware registrations now use the shared CDPTypes registry so local aliases, service-worker wire registration, and runtime validation stay in one place.

CDPTypes registry

  • Added js/src/types/CDPTypes.ts and translated equivalents in Python/Go.
  • The registry now manages:
    • generated native CDP command/event schemas,
    • built-in Mod.* schemas and service-worker expressions,
    • user custom command/event/middleware registrations,
    • JSON schema normalization for wire registration,
    • command parameter/result parsing,
    • event payload parsing,
    • generated imperative alias hydration.
  • Built-ins now include Mod.ping, Mod.configure, Mod.evaluate, Mod.getTopology, Mod.addCustomCommand, Mod.addCustomEvent, Mod.addMiddleware, and Mod.pong.
  • toJSON() omits expression bodies from serialized command/middleware config while still reporting registry state counts.

AutoSessionRouter and topology

  • Expanded AutoSessionRouter from target/session helper state into the owner of browser routing state:
    • sessionId_from_targetId
    • targetId_from_sessionId
    • targets
    • contexts
    • typed event subscriptions for Target.*, Runtime.*, and Page.* invalidation/update events.
  • Router startup now enables flattened auto-attach and target discovery through the selected UpstreamTransport.
  • Standard CDP methods are routed based on command metadata and current browser state:
    • browser-level domains go to the browser route,
    • target/session-bound commands resolve an addressable target route,
    • Runtime.callFunctionOn gets an execution context injected only when the browser-owned params do not already specify objectId, executionContextId, or uniqueContextId.
  • Added Mod.getTopology, which builds a single topology result containing:
    • objectGroup
    • rootFrameId
    • frames: { [frameId]: Frame }
    • roots: { [objectId]: DomRoot }
    • targets: { [targetId]: TargetInfo }
    • contexts: { [contextKey]: ExecutionContext }
  • Topology discovery resolves frame trees, OOPIF targets, frame owner backend nodes, document roots, execution contexts, and pierced shadow roots. Work is parallelized with a bounded queue of 8 topology tasks.

Server/downstream split

  • Replaced the old monolithic ModCDPServer.ts implementation with a TS-only service-worker server class that composes:
    • client: ModCDPClient for browser-target upstream access,
    • downstream: DownstreamTransportSet for SDK/client-facing downstream connections,
    • shared client.types for command/event/middleware registry state.
  • extension/src/service_worker.ts is now just service-worker bootstrap: instantiate new ModCDPServer() and await server.start().
  • Added TS-only downstream transport classes:
    • DownstreamTransport
    • DownstreamTransportSet
    • ReverseWSDownstreamTransport
    • NativeMessagingDownstreamTransport
    • NATSDownstreamTransport
  • Kept directionality explicit:
    • From ModCDPServer, downstream means SDK/client connections into the service worker.
    • From ModCDPServer, upstream means browser targets reached through server.client.
    • reversews is a server downstream transport and only a client-side upstream transport.
  • Added the server.downstream methods/semantics for request handlers, explicit responses, event fanout, client lease tracking, and downstream disconnect browser-close behavior.

Transport changes

Shared upstream surface

  • Consolidated translated upstream support around UpstreamTransport and WSUpstreamTransport.
  • Added a single send(...) surface on upstream transports with typed command support, event listeners, getTargets, createTarget, attachToTarget, and detachFromTarget helpers.
  • UpstreamTransport now exposes peer_kind so client-side transports that terminate at a ModCDP server peer can bypass local router bootstrap and forward CDP-shaped commands directly.

TS-only transports

  • Added/renamed TS-only transport files with explicit comments marking them as not translated:
    • ChromeDebuggerUpstreamTransport.ts
    • PipeUpstreamTransport.ts
    • ReverseWSUpstreamTransport.ts
    • NATSUpstreamTransport.ts
    • NativeMessagingUpstreamTransport.ts
    • downstream transport files listed above.
  • Renamed Nats* to NATS* and ReverseWebSocket* to ReverseWS*.
  • Removed Python/Go translations of exotic transports: pipe, nativemessaging, NATS, and reversews are no longer present in Python/Go source, tests, demos, or README content.

Config/naming cleanup

  • Replaced loose option bags and duplicated aliases with config objects validated at the owning boundary.
  • Standardized public config group names:
    • launcher
    • upstream
    • injector
    • router
    • client_config
    • server_config
    • types
  • Standardized mode names and file/class names:
    • bb / BBBrowserLauncher / BBExtensionInjector
    • cli / CLIExtensionInjector
    • cdp / CDPExtensionInjector
    • discover / DiscoverExtensionInjector
    • none / NoneBrowserLauncher
    • ws / WSUpstreamTransport
  • Cleaned up owner-prefixed config fields, for example:
    • launcher_local_*
    • launcher_remote_*
    • launcher_bb_*
    • injector_cli_*
    • injector_cdp_*
    • injector_bb_*
    • injector_discover_*
    • upstream_ws_*
  • Removed stale names such as NoopBrowserLauncher, BrowserbaseBrowserLauncher, ExtensionsLoadUnpackedInjector, LocalBrowserLaunchExtensionInjector, DiscoveredExtensionInjector, and translated borrow injection.

Cross-language parity

Python

  • Moved translated runtime config/state to Pydantic model-based config objects and .config reads.
  • Added Python CDPTypes, toJSON, topology shapes, router config, and updated ModCDP type models.
  • Renamed/added translated classes and tests to match TS naming:
    • BBBrowserLauncher
    • NoneBrowserLauncher
    • BBExtensionInjector
    • CLIExtensionInjector
    • CDPExtensionInjector
    • DiscoverExtensionInjector
    • WSUpstreamTransport
  • Removed Python exotic transport implementations/tests and stale reversews docs.
  • Updated Python demo to the supported websocket-only upstream surface and added topology/custom command/middleware/event checks.

Go

  • Added Go CDPTypes, toJSON, topology/config shapes, and translated router/client changes.
  • Renamed/added translated classes and tests to match TS naming:
    • BBBrowserLauncher
    • NoneBrowserLauncher
    • BBExtensionInjector
    • CLIExtensionInjector
    • CDPExtensionInjector
    • DiscoverExtensionInjector
    • WSUpstreamTransport
  • Removed Go exotic transport implementations/tests and translated borrow injection.
  • Updated Go demo to the supported websocket-only upstream surface and added topology/custom command/middleware/event checks.

Translation markers

  • Added consistent top-of-file comments to translated files and TS-only files.
  • Tests now carry translation comments indicating that names, cases, setup, and edge coverage should remain 1:1 across languages unless the file is explicitly TS-only or Go-only.

Generated artifacts

  • Regenerated/updated generated protocol surfaces, including:
    • js/src/types/generated/aliases.ts
    • js/src/types/generated/cdp.ts
    • js/src/types/generated/zod/**
    • Python/Go generated protocol schema surfaces.
  • Refreshed extension artifacts:
    • python/modcdp/extension.zip
    • go/modcdp/injector/extension.zip
  • Added shared browser-path helper usage for examples/tests.

Extension UI/service-worker changes

  • Simplified the extension service worker to delegate to ModCDPServer instead of duplicating server bookkeeping in extension/src/service_worker.ts.
  • Added offscreen keepalive comments/TS-only marking.
  • Reworked the options page to render the structured toJSON() tree with config/state/children instead of dumping a hand-built service-worker status object.

Tests and CI

  • Updated CI matrices so Python and Go only run supported websocket upstream modes, while TS-only reversews/NATS/proxy paths stay in JS-specific jobs.
  • Added/renamed tests for the new class names and supported runtime surfaces.
  • Removed translated tests for unsupported Python/Go exotic transports.
  • Added topology and CDPTypes schema-normalization coverage across translated runtimes.
  • Added/updated real-browser demos/tests for custom command registration, custom events, middleware, ping/pong latency, and Mod.getTopology.

Validation run on the current head:

  • pnpm run typecheck
  • pnpm run build:package
  • pnpm exec prek run oxfmt --all-files
  • pnpm exec vitest run js/test/test.ReverseWSUpstreamTransport.ts --fileParallelism=false --maxWorkers=1
  • pnpm exec vitest run js/test/test.proxy.ts --testNamePattern "reversews" --fileParallelism=false --maxWorkers=1
  • cd python && uv run pyright
  • cd python && uv run ty check
  • cd go && go test -count=1 ./modcdp ./modcdp/transport

The current PR check suite was green on head cb4dc17, including the JS/Python/Go test/demo matrix, proxy examples, serialized reversews/NATS JS jobs, and prek.

Review guide

Suggested review order:

  1. js/src/types/modcdp.ts and js/src/types/CDPTypes.ts for public shapes and registry behavior.
  2. js/src/transport/* for upstream/downstream responsibility boundaries.
  3. js/src/router/AutoSessionRouter.ts for target/session/context/topology routing.
  4. js/src/server/ModCDPServer.ts and extension/src/service_worker.ts for the service-worker composition change.
  5. js/src/client/ModCDPClient.ts for constructor/config/send lifecycle changes.
  6. Python/Go translated siblings and tests to verify they match the TS files marked MODCDP_TRANSLATE and omit TS-only files.

@pirate pirate marked this pull request as ready for review May 26, 2026 19:37
@pirate pirate merged commit a2466b2 into main May 28, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants